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Abstract 


For many reasons the use of computer-assisted assessment (CAA) is increasing. 
Although computer-based examinations increase in use, research is lacking about 
students' perceptions of online assessment in general and of categorized fields of 
online assessment systems. The aim of the study was to investigate students' 
perceptions of the use of CAA and to investigate the potential for using student 
feedback in the validation of assessment. To determine the students' perceptions of 
online assessment, an examination Web site was developed and implemented as 
part of the assessment of Masalistu Yayincilik (Desktop Publishing), a course given 
by the Department of Computer Science at Kocaeli University, Turkey. The study 
was descriptive, using a paper-based survey and interviews for data collection. 
Participants were third-year students enrolled in the course. Descriptive analysis 
of the questionnaire and interview data showed that the most prominent features 
of the online assessment system were immediate feedback, randomized question 
order, item analysis of the questions, and obtaining the scores immediately after 
the exam. Participants reported the effectiveness of the online assessment system. 
Although there is much room for improvement in online assessment systems in the 
near future, such systems are accepted by computer-friendly youth. 

Resume 


L'utilisation de revaluation assistee par ordinateur est en hausse pour plusieurs 
raisons. Bien que l'emploi d'examens par ordinateur soit en croissance, la re- 
cherche fait defaut sur la perception qu'ont les etudiants de ces derniers et des 
categories de systemes devaluation en ligne. Le but de l'etude etait d'examiner 
comment les etudiants pergoivent revaluation assistee par ordinateur et d'evaluer 
comment les reactions des etudiants peuvent guider la recherche a ce chapitre. 
Afin de determiner les perceptions des etudiants a propos des evaluations en ligne, 
un site Web d'examens a ete congu dans le cadre de 1'evaluation du cours Masaiistii 
Yayincilik (Publication assistee par ordinateur), donne par le departement des 
sciences informatiques a Kocaeli University en Turquie. L'etude, de nature des- 
criptive, utilisait un questionnaire papier ainsi que des entrevues pour recueillir les 
donnees. Les participants etaient des etudiants de 3 e annee inscrits a ce cours. Une 
analyse descriptive des questionnaires et des donnees recoltees en entrevue a 
revele que les caracteristiques dominantes du systeme d'evaluation en ligne etaient 
la retroaction immediate, la repartition aleatoire des questions, l'analyse de chaque 



78 


M.Y. OZDEN, I. ERTURK, and R. SANLI 


element des questions et l'obtention immediate du resultat de l'examen. Les parti- 
cipants ont note l'efficacite du systeme devaluation en ligne. Meme s'il y a ample- 
ment place a amelioration des systemes devaluation en ligne, on remarque que ces 
derniers sont tout de meme deja acceptes par les jeunes de la generation informa- 
tique. 


Background of the Study 

In recent years developments in information and communication tech- 
nologies (ICT) have led to growth in the range of Internet tools that can be 
used for learning and research. Some have gained wide-scale acceptance 
(e.g., the ease with which e-mail has been adopted); others seem to find 
either niche applications or are less pervasive than one might at first have 
imagined (e.g., videoconferencing). One application that is becoming 
more common is computer-assisted assessment. The term computer-assisted 
assessment can cover any kind of computer use in the process of assessing 
knowledge, skills, and abilities of individuals. 

Computer-assisted assessment (CAA) encompasses a range of ac- 
tivities, including the delivery, marking, and analysis of all or part of the 
student assessment process using stand-alone or networked computers 
and associated technologies. Earlier research has shown a range of motiva- 
tions for implementing CAA in a course, and often a combination of 
factors result in CAA being used (Bull & McKenna, 2001). Some of the key 
reasons cited include: 

• To increase the frequency of assessment, motivating students to learn 
and encouraging skills practice; 

• To broaden the range of knowledge assessed; 

• To increase feedback to students and lecturers; 

• To extend the range of assessment methods; 

• To increase objectivity and consistency; 

• To reduce marking loads; and 

• To aid administrative efficiency. 

This article describes the findings of an external evaluation of a project 
that aimed to disseminate good practice, guidelines, and models of im- 
plementation and evaluation of one particular type of learning technol- 
ogy, namely, CAA. In particular the evaluation explored the effect of 
integrating CAA in learning and teaching and the perception of students 
about CAA. 

Purpose of the Study 

Over the past decade there has been a large increase in the use of com- 
puter-based assessment (Stephens & Mascia, 1997). However, little has 
been published to date on students' views of computer-based assessment. 
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particularly that based on more complex interactions offered by the 
TRIADs system (Mackenzie, 1997). Because some of the published works 
are on the prevalence of computer anxiety among students, the use of 
computers for assessment has been open to question. This comes with a 
general recognition in higher education that this assessment is no longer 
separate from, but rather affects, all stages of the learning process (Brown 
& Knight, 1995). Given the history of CAA, we were interested in observ- 
ing the effect of the introduction of CAA on the learning process and to 
investigate further the perception of students. The aim of the study was to 
gain an understanding of students' perceptions of the use of CAA and to 
investigate the potential for using student feedback in the validation of 
assessment. 

Significance of the Study 

The use of computer-based assessment is increasing for many reasons. 
Examples include entrance exams in education, military training exams, 
and certification exams by professional groups. Although the use of com- 
puter-based exams is increasing, there is not enough research about 
students' perceptions of online assessment in general and of categorized 
fields of online assessment systems. Such research would give detailed 
information about which parts of the online assessment systems are im- 
portant or which parts of the systems should be developed or revised to 
achieve better results. 

Design of the Study 

The descriptive study used paper-based surveys and interviews for data 
collection. To obtain information about the students' perceptions of online 
assessment, a Web site was developed and implemented. The course in- 
structor was responsible for the instructional design, content creation, and 
all activities for the course, but one researcher designed and developed the 
online assessment Web site. The Web site was database driven and devel- 
oped using Active Server Pages (ASP), a Microsoft Access Database, and 
Cascading Style Sheets (CSS). The online assessment Web site was mainly 
designed as two modules with user and administrator interfaces. The user 
module contained multiple-choice questions. This online assessment site 
was used for the module assessment part of Masaiistii Yayincilik (Desktop 
Publishing), a computer course given by the Department of Computer 
Education at Kocaeli University in spring 2003 about computer literacy, 
MS Office applications, and Web development tools. The instructor of the 
course was not allowed to see the data obtained from questionnaires and 
interviews before giving the final marks. 
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Table 1 

Demographic Characteristics of Participants 



N (46) 

% 

Sex 



Male 

40 

87 

Female 

6 

13 

Age 



20 

6 

13 

21 

30 

65 

22 

10 

21 


Participants 

Participants were third-year students in the Department of Computer 
Education, Kocaeli University, enrolled in the course Masaristu Yayincilik 
(Desktop Publishing). All the students taking the course were informed 
about the research, and all were free to choose whether to complete the 
questionnaires and attend interviews. In the study all students showed a 
strong willingness to participate in the research. Forty-six students 
registered in the course and their demographic characteristics are shown 
in Table 1. 

Instrumentation 

A paper-based questionnaire and in-depth interviews were used to inves- 
tigate the students' perceptions of the online assessment. Each tool used in 
this study is described below. 

User evaluation questionnaire. This questionnaire was designed to obtain 
information on the students' computer familiarity and prior online assess- 
ment experience and their evaluation of specific components such as user 
interface, effects on the learning process, and system usage of the online 
assessment web site. Two measurement and evaluation experts and one 
distance education expert from the Faculty of Education, METU contrib- 
uted to preparing the questionnaire. The questions were of three types: 
nominal data responses; Likert five-point scale items from strongly agree, 
agree, neutral, disagree, to strongly disagree; and open-ended responses. 

In-depth interviews. In order to gain a better understanding of the re- 
sponses and suggestions for the online assessment system, especially in 
terms of its function design, implementation, and Web site production, we 
decided to supplement the study with follow-up interviews. In-depth 
interviews probed the opinions and suggestions of the users about the 
unresolved answers and controversial issues that were not revealed 
through the earlier questionnaire. A researcher design interview protocol 
was developed after the survey data were analyzed (see Appendix). 
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Reliability 

Reliability refers to the consistency of responses over time. In order to 
assess the reliability of this questionnaire, a pilot study was undertaken 
with five students randomly chosen from the population. Cronbach's 
alpha coefficient was calculated from the results of this pilot study. The 
resulting scores were all at least 0.75. Data from the open-ended responses 
were used to improve the Web design. Subsequently, Cronbach's alpha 
coefficients were also assessed for the actual study responses, and all 
scores were at least 0.75, showing that the reliability level of this research 
was consistently high. 

Data Collection and Analysis 

The evaluation survey was conducted one and a half months after the 
course began. The paper-based questionnaire was distributed to a class of 
46 students who were taking the course. The responses were collated and 
percentages and mean values calculated. 

After the course was completed, in-depth individual interviews were 
conducted with five randomly selected students using a random number 
sequence. During the interviews students' responses were written down. 
Each interview took almost half an hour. The data were collated and 
responses linked to the numerical survey data and the open responses. In 
addition, usage data for the Web site were printed. The course instructor 
was not allowed to see the data obtained from questionnaires and inter- 
views before giving the final marks. 

Findings 

Question 1: What was the participants’ competence with computers? 

The purpose of this question was to investigate the students' competence 
with computer application programs such as Web browsers and e-mail 
programs as being familiar with those programs is a prerequisite for using 
the Web-assisted assessment program (Table 2). 

Four percent of all students indicated that their competence level with 
the Web browser was poor. For effective use of the developed online 
assessment tool, it was enough to have an introductory competence level. 
The total percentage of the students beyond the introductory competence 
level was 96%. Before the final examination students were given sample 
quizzes and trained on the important points of the online assessment tool. 
Thus any problem caused by browser usage was eliminated. 

Question 2: What were the prior experiences of participants for online 
assessment? 

The purpose of this question was to identify the students' prior experience 
with online assessment (see Table 3). 
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Table 2 

Percentage Distribution of Students’ Computer Competence (A/=46) 


Student Competence with 

Advanced 

% 

Good 

% 

introductory 

% 

Poor 

% 

None 

1 . Web browser 

25 

54 

17 

4 

0 

2. Chat 

25 

34 

29 

8 

4 

3. Telnet 

21 

37 

17 

0 

25 

4. E-mail 

25 

45 

13 

13 

4 

5. FTP 

8 

46 

21 

0 

25 

5. Mailing lists 

33 

38 

8 

4 

17 


As the results indicated, most of the students had no prior experience 
with online assessment. Only 42% had used the Internet for instructional 
purposes, 20% had taken online Web quizzes, and 33% had previously 
taken some kind of online assessment. Unfortunately, none of the students 
had taken an online course before the research. 

Question 3: What were participants’ perceptions of the user interface of 
the online assessment Web site? 

Table 4 shows the evaluation items of the system in terms of screens and 
interface. The agreeability mean of the users is also high at over 3.75. 
Almost all standard deviations are less than 1.00. These indicate that 
almost all users have common thoughts in terms of the user interface. 
Based on the results of our survey, the appropriateness in terms of the 
overall framework, the overall configuration of colors and background, 
the overall layout of screen and window design, and overall interface 
operation method were appraised highly. In addition, the appraisal of the 
appropriateness of screen design and ease of use of the interface operation 
were both scored highly and evenly. 

Whereas 59% of users saw the help page interface as clear and easy to 
operate, 33% disagreed. In terms of the standard deviations, this item 


Table 3 

Students’ Prior Experiences of Online Assessment (A/=46) 


Prior experiences 

Yes % 

No % 

1 am taking course(s) online 

0 

100 

1 have attended an online course before 

0 

100 

1 have taken TOEFL or GRE before 

0 

100 

1 have taken some kind of online assessments before 

33 

67 

1 have taken an online quiz on the web 

20 

80 

1 have used Web for instructional purposes 

42 

58 
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Table 4 

Frequencies, Percentages and Means of Student Agreement in Online 
Assessment System “Screen and Interface Design” 


Evaluation of User Perception 

Percentages of Agreement (%) 

1 



of Online Assessment 

Frequency Distribution 



Means 

SD 


5 

4 

3 

2 

1 



User interface evaluation 
1 . Overall framework and 
operation levels of the system 
are clear and smooth 

23 

35 

38 

4 

0 

3.77 

0.86 


11 

16 

17 

2 

0 



2. Overall configuration color 
and background is normal 
harmonious for the system 

14 

55 

27 

4 

0 

3.79 

0.75 


6 

26 

12 

2 

0 



3. Overall screen layout and 
window design of the system 
is appropriate 

18 

64 

14 

4 

0 

3.96 

0.72 


8 

30 

6 

2 

0 



4. Overall interface operation 
method is easy and 
appropriate 

17 

35 

35 

9 

4 

3.52 

1.03 


8 

16 

16 

4 

2 



5. Log-in interface is clear and 
easy to operate 

22 

35 

30 

9 

4 

3.62 

1.07 


10 

16 

14 

4 

2 



6. Log-in interface design is 
appropriate 

22 

52 

17 

9 

0 

3.87 

0.86 


10 

24 

8 

4 

0 



7. Register interface is clear 
and easy to operate 

23 

30 

43 

4 

0 

3.72 

0.87 


11 

13 

20 

2 

0 



8. Register interface design is 
appropriate 

18 

48 

30 

4 

0 

3.80 

0.79 


8 

22 

14 

2 

0 



9. Exam interface is clear and 
easy to operate 

27 

39 

30 

4 

0 

3.89 

0.86 


12 

18 

14 

2 

0 



10. Exam interface design is 
appropriate 

18 

43 

30 

9 

0 

3.70 

0.87 


8 

20 

14 

4 

0 



1 1 . Past exam results 
interface is clear and easy to 
operate 

17 

58 

17 

8 

0 

3.84 

0.81 


8 

26 

8 

4 

0 



12. Past exam results 
interface design is appropriate 

26 

29 


338 

4 

3.65 

1.09 


12 

13 

15 

4 

2 
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Table 4 (continued) 


Evaluation of User Perception 

Percentages of Agreement (% 

) 



towards Online Assessment 

Frequency Distribution 



Means 

SD 


5 

4 

3 

2 

1 



User interface evaluation 
13. Statistical evaluation 
interface is clear and easy to 
operate 

16 

50 

25 

9 

0 

3.73 

0.87 


7 

23 

12 

4 

0 



14. Statistical evaluation 
interface design is appropriate 

17 

46 

33 

4 

0 

3.76 

0.79 


8 

21 

15 

2 

0 



15. Exam result interface is 
clear and easy to operate 

29 

42 

25 

4 

0 

3.96 

0.93 


13 

19 

12 

2 

0 



16. Exam result interface 
design is appropriate 

13 

50 

29 

8 

0 

3.68 

0.81 


6 

23 

13 

4 

0 



17. Help page interface is 
clear and easy to operate 

21 

38 

11 

13 

17 

3.33 

1.38 


10 

17 

5 

6 

8 



18. Help page interface design 
is appropriate 

37 

38 

16 

9 

0 

4.03 

1.11 


17 

17 

8 

4 

0 




showed the biggest standard deviation at 1.38. It also had the smallest 
mean score (3.33). All this shows that in terms of the help page, users did 
not agree as to whether it was good or bad, but the trend is negative 
relative to the other items in the questionnaire. Although all the users were 
asked to read the help page, usage data indicated that most did not read it 
but went directly to the exam pages. Thus although some applicants said 
that the help page interface was not clear and easy to operate, it is likely 
that they did not read it. Therefore, help pages that are more effective and 
easy to use should be provided to meet the learners' needs more effective- 
ly. Help pages should encourage the participants to read while they use 
the online assessment tools. In contrast to the above, almost all users 
indicated that the help page interface design was appropriate. The mean 
value for this questionnaire item was 4.03 and standard deviation was 
1.11. This suggests that the interface design of the help page was good, but 
not good enough to use. 

A majority of students (53-71%) rated the various interfaces as clear 
and easy to operate. Negative responses ranged from 4% to 13%, with the 
highest number being about the log-in interface. Responses about the 
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clarity and operability of the register interface resulted in the largest per- 
centage (43%) of neutral responses. Higher percentages of students (63- 
75%) rated the interfaces as appropriate. The exception was the past exam 
results interface, with 12% giving negative ratings. 

These results may be explained by students' infrequent use of this part 
of the online assessment tool, which may have led to their negative 
opinions. Although the mean values in the user interface evaluation are 
over 3.50, there is room for improvement 

Question 4: What are participants’ perceptions about system use of the 
online assessment Web site? 

The aspects of system use are shown in Table 5. The means were between 
3.50 and 4.17, and the standard deviation of most questions was less than 
1. This shows that users used the Web pages with no significant problems. 
Problems with the use of the help page resurfaced, with 54% indicating 
positive and 17% negative support for the statement "Help page made me 
use the Web site better." On average, 71% of users agreed that browsing 
the Web pages was easy, and 75% agreed that directions were followed 
with no problem; registration to the system and taking the exam were 
easy; the system was easy to use and comfortable; and changes could be 
made easily. 

It is likely that the high scores on these items may be a result of the 
initial training on system use and sample quizzes taken before the final 
exam. 

Question 5: What are participants’ perceptions about the impact of the 
online assessment Web site on the learning process? 

Questions were asked about three topics: assessment, cheating, and use 
(see Table 6). In terms of the fairness of the assessment process, 74% rated 
it positively, whereas 10 students (21%) were uncertain. However, when 
asked to respond to the statement "Cheating is difficult," the majority 
(54%) of students disagreed whereas only 33% agreed. To prevent cheat- 
ing in the system, questions were asked in random order, and placement 
of the options of the questions were also varied from user to user. In 
addition, all exams were taken in the labs under the supervision of proc- 
tors. Students may not have been aware of these procedures. At least 70% 
of students thought that the system feedback helped them reflect on their 
learning, and page-by-page questions made them feel better in the exam. 
Most students thought their own growth had improved through use of the 
system, and 67% hoped to see the system used in the other courses. 
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Table 5 

Frequencies, Percentages and Means of Student Agreement in Online 
Assessment System “System Use” 


Evaluation of User Perception 

Percentages of Agreement (%) 

1 



towards Online Assessment 

Frequency Distribution 



Means 

SD 


5 

4 

3 

2 

1 



System Use 

1. 1 have browsed among Web 
pages easily 

21 

50 

25 

4 

0 

3.88 

0.82 


10 

22 

12 

2 

0 



2. 1 have followed the direction 
without any problem 

46 

29 

21 

4 

0 

4.17 

0.91 


21 

13 

10 

2 

0 



3. It is easy to register to 
system 

46 

29 

21 

4 

0 

4.17 

1.05 


21 

13 

10 

2 

0 



4. It is easy to take an exam 

47 

29 

20 

4 

0 

4.19 

0.92 


22 

13 

9 

2 

0 



5. Easier to correct work 

31 

46 

19 

4 

0 

4.04 

0.79 


14 

21 

9 

2 

0 



6. Ease of use and comfortable 

37 

38 

21 

4 

0 

4.08 

0.88 


17 

17 

10 

2 

0 



7. 1 often visit the past exam 
result page 

28 

42 

17 

13 

0 

3.85 

0.92 


13 

19 

8 

6 

0 



8. Help page made me use the 
Web site better 

16 

38 

29 

17 

0 

3.53 

1.97 


7 

18 

13 

8 

0 



9. Seeing left time makes me 
progress better 

38 

33 

21 

4 

4 

3.97 

1.07 


17 

15 

10 

2 

2 




Question 6: What are the participants’ opinions about the online 
assessment Web site? 

Students' general opinionss about the online assessment tool were also 
investigated. As shown in Table 7, 58% of users agreed that the system 
provided immediate feedback, 79% agreed that online assessment was 
better than the paper-and-pencil format, and 92% agreed that online as- 
sessment was faster than the paper-and-pencil form. On average, 80% of 
the users agreed that online assessment was contemporary and more 
systematic. All the users thought this kind of the online assessment was 
consistent with the teaching style, but 30% disagreed that they were less 
anxious. 
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Table 6 

Frequencies, Percentages, and Means of Student Agreement about 
“Impacts on Learning Process” 


Evaluation of User Perception 

Percentages of Agreement (% 

) 



towards Online Assessment 

Frequency Distribution 



Means 

SD 


5 

4 

3 

2 

1 



Impacts on learning process 
1 . Assessment is fair 

41 

33 

21 

5 

0 

4.10 

0.97 


19 

15 

10 

2 

0 



2. Cheating is difficult 

20 

13 

13 

33 

21 

2.78 

1.47 


9 

6 

6 

15 

10 



3. System feedback helps me 
to reflect on my merits in 
learning 

18 

64 

18 

0 

0 

4.00 

0.61 


8 

30 

8 

0 

0 



4. Tracking past exam results 
makes me understand my 
progress 

41 

36 

13 

5 

5 

4.03 

1.04 


19 

17 

6 

2 

2 



5. Statistical evaluation page 
gives a detailed information on 
units where 1 am good or 
unsuccessful 

18 

30 

39 

13 

0 

3.53 

0.94 


8 

14 

18 

6 

0 



6. It helps me to better 
understand my growth and 
improvements in the course by 
using the system 

13 

50 

33 

4 

0 

3.72 

0.75 


6 

23 

15 

2 

0 



7. It helps me to learn this 
course by using this system 

21 

42 

29 

8 

0 

3.76 

0.89 


10 

19 

13 

4 

0 



8. 1 hope to use this system in 
other courses as well 

38 

29 

25 

4 

4 

3.93 

1.03 


17 

13 

12 

2 

2 



9. Page-by-page questions 
makes me feel better in the 








exam 

37 

33 

13 

17 

0 

3.90 

1.07 


17 

15 

6 

8 

0 




Results of In-Depth Interviews with Users 

After analyzing the results, we conducted in-depth interviews with five 
users who were chosen randomly from the students who had been taking 
the course. 

Regarding the system function, some students viewed the exam style 
as inconvenient because the questions were selected randomly from a 
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Table 7 

Percentages and Means of Student Agreement in Online Assessment System 
“Student opinions” 


Evaluation of User Perception 

Percentages of Agreement (% 

) 



towards Online Assessment 

Frequency Distribution 



Means 

SD 


5 

4 

3 

2 

1 



Students’ Opinions 

1 . System provides immediate 

feedback 

12 

46 

38 

4 

0 

3.66 

0.76 


6 

21 

17 

2 

0 



2. Less anxious 

15 

38 

17 

13 

17 

3.21 

1.35 


7 

17 

8 

6 

8 



3. Better than 
paper-and-pencil form 

71 

8 

13 

4 

4 

4.38 

1.13 


32 

4 

6 

2 

2 



4. Consistent with the teaching 
style 

25 

42 

33 

0 

0 

3.92 

0.77 


12 

19 

15 

0 

0 



5. Faster than paper-and-pencil 

59 

33 

4 

0 

4 

4.43 

0.92 


27 

15 

2 

0 

2 



6. Contemporary 

62 

17 

17 

0 

4 

4.33 

1.04 


28 

8 

8 

0 

2 



7. More systematic 

37 

38 

21 

0 

4 

4.04 

0.99 


17 

17 

10 

0 

2 



8. Can be applied to other 








courses 

24 

50 

13 

13 

0 

3.85 

0.94 


11 

23 

6 

6 

0 




question pool. They suggested that questions should appear in the form of 
ordered categories and that questions in this category should appear ran- 
domly on the exam screen. 

Some students suggested we add a notebook area. This would allow 
students to keep their notes permanently to use whenever they wished. 
This would positively affect their learning process. 

Another problem discovered during the interviews was that students 
could not see their selections on the completed exam pages although they 
could see all the exam pages. They suggested that when they visited a 
completed exam page, they should be able to see their selections there to 
enable them to make any changes easily. Prior answers were hidden to 
prevent cheating. Based on the interview data, this should be reconsidered 
as a way of improving the effectiveness of the online assessment tool. 
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Discussion 

The purpose of this study was to investigate students' perceptions about 
the use of online assessment. A Web site and exam system were used for 
summative assessment of computer education students for Masaiistu 
Yayincilik (Desktop Publishing), a course given in the spring term of the 
2003-2004 academic year at Kocaeli University. 

Descriptive analysis of the questionnaire and interviews showed that 
the most prominent features of the system were immediate feedback, 
randomized question order, item analysis of the questions, and obtaining 
the scores immediately after the exam. Overall, participants agreed on the 
effectiveness of the online assessment system. 

Most of the students agreed that the features of obtaining immediate 
scores and feedback motivated them and contributed positively to their 
achievement on the exam. These features are the main advantages of 
computer-based compared with paper-based exams. The greatest physical 
differences between computer and paper test administration are perceived 
as the interactivity and physical size of the display area. The amount of 
information comfortably presented in a computer display is only about 
one third of that presented by a standard sheet of paper. For example, 
Haas and Hayes (1986) reported that when a text passage associated with 
a test item required more than one page, computer administration yielded 
lower scores than paper-and-pencil administration, apparently because of 
the difficulty of reading the extended text on-screen. A student can rapidly 
scan all the questions on a page and can easily flip backward or forward 
to other pages (a form of interactivity). In computer-based assessment, one 
test item is presented on each computer screen display, and the student 
needs to act physically to move from screen (item) to screen (another form 
of interactivity). This difference probably leads to greater focus and 
closure with each computer-based item. Thus computer-based items (rela- 
tive to paper) may increase transition time and memory load with a tighter 
focus on and closure of each individual item (Clariana, 1997). 

Some students also suggested that units of the subjects should appear 
in an ascending order, but that the questions in the units should appear 
randomly. Item order (computer-administered test items are presented in 
a randomized order) and the order of multiple-choice response options 
(randomized in computer administered tests) can affect performance on 
an item (Beaton & Zwick, 1990). This probably relates to ordered versus 
randomized test item sequencing. Specifically, when the instructional les- 
son content and the test items are in the same order, the ordered test will 
probably obtain greater scores than a randomized version. In our inves- 
tigation, the computer-based tests were randomly generated, thus justify- 
ing an order effect. 
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The other valued features were simplicity of testing, comfort, speed, 
simplicity of editing and alterations, effective measure of learning out- 
comes, and reduced anxiety (Karakaya, 2001). 

Both high- and low-ability students should benefit from greater focus 
on an item, although because of the greater cognitive load required, only 
high-ability students would be able to tie ongoing items together to learn 
from the test in order to answer other test items. To examine this hypothe- 
sis, a test could be designed that intentionally provides items that, if 
remembered, would allow the student to answer other items correctly. If 
high-ability learners do learn during the test (relatively), a pattern of 
means similar to that observed in this investigation should occur. If dis- 
play-size format is the primary factor, then the multiple-page group 
should outperform the one-item-per-page format. 

Conclusion 

Based on our review and study results, we anticipate that computer and 
assessment tool familiarity are the most fundamental key factors in the 
perception of online assessment, especially for unfamiliar content and/or 
for low-attaining examinees (especially an issue for students with reduced 
computer access such as women and minorities). In general, higher-attain- 
ing students will adapt most quickly to any new assessment approach 
(Watson, 2001) and will quickly develop test-taking strategies that benefit 
from the new approach. Thus in the current investigation, because stu- 
dents are from the Department of Computer Education, the higher-attain- 
ing students probably accommodated more quickly and so benefited more 
from computer-based assessment. Once all students are fully familiar with 
computers, familiarity should become less important. 

Although students were trained before the exam about how to use the 
online assessment system, some felt anxious in the exam. In order to 
prevent such problems, students must be comfortable with the online 
assessment system, and the context in which they are taking the exam 
should have a warm atmosphere. 

Using online assessment requires close cooperation of academic and 
technical units. First, preparing questions for online settings requires extra 
effort. Questions should measure the intended level of knowledge. In- 
structors should be trained on how to conduct a course online and ask 
questions via the Internet. Administrative units should support such a 
teaching-learning environment and should prepare the required structure 
for the system. Finally, this type of assessment system works through 
technological devices: computers, network devices, and so forth. Com- 
puters must be powerful enough to run the Web pages, and the server 
should be stable. 
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Bugbee (1996) recommends that test developers show that computer- 
based and paper-based test versions are equivalent and/or must provide 
scaling information to allow the two to be equated. Most instructors, and 
in fact even most instructional designers, do not have the skill or the time 
and expertise to pilot their examinations extensively. However, additional 
time and effort must be invested by instructors to design quality test items 
for use in online testing. With the likely production of Web-based courses 
and of inexpensive fingerprint identification computer devices and other 
automatic supervisory technologies, computer-based testing will probably 
increase substantially. The findings of this investigation indicate that it is 
critical to realize that computer-based tests, even with identical items, will 
not necessarily produce equivalent measures of student learning. Instruc- 
tors and institutions should spend the time, money, and effort to create 
positive student perception of online assessment. 
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Appendix 

Student Interview Questions 

Which component or area needs to be improved most? 

Is the screen and interface design of this online assessment system appro- 
priate and convenient to use? 

Is the system use of this online assessment system easy to use? 

Does the online assessment system have a positive effect on learning 
progress? 

What are the difficulties faced while using the online assessment system? 
What did you like most while using the online assessment system? 

Is there any other issue or area that has not been mentioned in the ques- 
tionnaire but need to be improved? 



